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ABSTBACT 

Program evaluation can be used for both program 
improvement and program justification, and it can aid in program 
planning and policy makdlng; In evaluation, we must be concerned with 
rhe components which not only make up the program but also the 
evaluation such as program objectives or goals, program personnel 
interaction, program processes, resources, intended audiences, 
evaluation objectives, types and levels of evaluation data, 
judgment- ma king processes, evaluator roles and responsibilities, etc. 
The process of evaluation has no single set of procedures, but the 
similarities among evaluation models can be identified as: program 
description; criteria formulation; evidence gathering; judgment 
making; and dissemination. Established and potential rural 
development evaluators must be aware of the pitfalls inherent in tjie 
evaluation process; the following are some of the evaluator |s 
concerns: irreproachable ,ethics; program evaluation responsibility 
vested in a single person; communication of all evaluation plans to 
the total program staff; data derived from many rather then one 
source; sensitivity to the necessity of diplomacy; areas of 
negotiation by program staff and evaluator; flexibility; spelling out 
program options rather than making actual decisions; commitment to 
evaluation; evaluation feedback to the proper personnel; and 
solicitation of feedback prior to implementation. (JC) 
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Preface 



This manuscript was prepared as a response to the rural development 
• practicioner 's need for knowledge on the subject of evaluative research. 
However, this paper may best be viewed as just the tip of the evaluation 
iceberg. Hopefully, I have included mos t of the relevant concerns for 
evaluation from the practicioner's standpoint. In my range of exposure 
to rural development and evaluation, and most notably, the some 18 
months of involvement as Co-Leader of this ''Synthesis'* Project, -I. have 
come into contact with some of the most interesting and knowledgeable 
persons in the areas of rural development and evaluation. The adminis- 
trators, researchers, practicioners, educators and "Functional Network" 
members who supplied inputs to this project will probably find their ideas 
and/or philosophies co-opted for this docume t. This, I believe, is 
good since the purpose of this paper and of the Project as a whole is 
svnthesis. 



G. Richard Wetherill 
August, 1976 



Evaluation and Rural Development 



Network Members 



Mr. Dean E. Barrett 

Oklahoma State Universitv 



Dr. Freddie Richards 

Prairie View A&M University 



Dr. Lawrence Clinton 

East Texas State University 



Dr. Andres Steinmetz 

University of Virginia 



Dr. Arthur G. Cosby, Center Associate 
Texas A&M Universitv 



Mr. G. Richard Wetherill 
Texas A&M University 



Dr. Harold B. Crawford 

Program Evaluator at Lar^e 



Dr. Gerald C. Whaelock 
Alabama A&M University 



Mr. Howard Ladewig 

North Carolina State University 



3 



THE "PROCESS OF EVALUATION IN RURAL DEVELOPMENT 



Introduction 

The rather brief history of evaluation in rural development has been 
characterized by an attitude of foreboding. This seems to be one of the most 
prevalent problems in this area at the present time. Such an attitude appears 
to stem mainly from a lack of information regarding what principles of evalua- 
tion are applicable to rural development. The undergirding purpose of this 
paper is that of providing rural development practitioners with a brief sketch 
of the tools necessary to conduct an evaluation of rural development programs. 
Four key issues will be dealt with in this paper: (1) the purpose of evalua- 
tion, (2) what we evaluate, (3) the evaluation process, and (4) the problems 
and pitfalls in evaluation. Although these four key issues cannot by any 
stretch of the imagination be considered as the total range of the evaluation 
process, they may be viewed as four issues which any potential evaluator must 
address before getting started in evaluative research. 

There are many definitions of evaluation. Much of the controversy in 
the field today concerns the development of an adequate definition for evalua- 
tion (Ladewig, 1976). Many of these controversies spring from definitions of 
evaluation which lend a threatening connotation to evaluation. Perhaps what 
would be most beneficial for the evaluation of rural development programs, 
would be a definition of evaluation which is non-threatening in nature. One 
frame in which such an evaluation definition could be couched is one which 
results from the definition dichotomy of "evaluation for program improvement" 
versus "evaluation for program justification." The most non-threatening 
definitions of evaluation tend to come from those who define evaluation as 
being for the primary purpose of program improvement, while the more threat- 
ening definitions relate directly to justification of programs. Therefore, 
for our purposes h6re, evaluation may be defined as the systematic examination 
of a program in operation for the purpose of improvement (Wetherill and Buttram, 
1976). This process takes the form of assessing a program in terms of its 
stated goals and objectives. Such evaluative information is then used in 
order to make decisions concerning program improvement. Although such a defini- 
tion may. be limited and rather elementary, it can serve as a useful starting 
point for beginning an evaluation. 

Evaluation research may be considered more applied than "pure." Ev^n 
with all things considered, the ranges of evaluation are great. In terms of 
"things" evaluated, the range may run the gamut from classroom/ teacher-type 
evaluations, through detailed program evaluations, all the way up to the 
evaluation of social subsystems. In terms of the degree of evaluative rigor, 
the range may run from cerebral type of evaluations through formal evaluations. 
The techniques of evaluation research alf:o encompass a wide range, including 
informal techniques (participant observations or other unobtrusive measures) 
up to and including experimental and quasi-experimental designs. It is 
usually up to the evaluator and/or program staff and adrfeinistrators to determine 
wherein these ranges the evaluation may be specified * Such things roust be 
considered in developing a personal philor-:ophy of evaluation. 
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Why Evaluate?; The Purpose of Evaluation 



Everyone evaluates. Our daily lives are filled with decisions we must 

make based on the conclusions which we draw from the data that we receive, 

This informal evaluation, more often than not non-verbalized, is part and 

parcel of our daily living* 

In the case of project or program evaluation, all too often informal 
evaluation is the only type that has been done- Given the increased national 
emphasis for requiring program evaluation (Hammil, 1975), what is needed now 
is a formalization of the ofttimes informal process of evaluation. The 
benefits of such formalization are potentially tremendous. At the very least, 
evaluation may be used to provide documentation concerning the history and/or 
impacts of a program. Better evaluations will provide program decision makers 
with the necessary evaluation information in order to help them make better 
decisions toward better programs. In this light, program evaluation can be 
used for both program improvement and program justification, and provide for 
program planning and policy making. The fact remains that, like it or not, 
program evaluation is fast becoming a mandatory part of program operations 
(Wholey, et, al, , 1973), 

What Do We Evaluate? 

The answer to the question posed by the title of this section is not 
simply ans\>rered by saying, "The program," In evaluation, we must be concerned 
with the components which not only make up the program, but also make up the 
evaluation. Here, we mean such things as: program objectives or goals, 
program personnel interaction^ program processes, resources, intended audiences, 
evaluation objectives, types and levels of evaluation data, judgment-making 
processes, evaluator roles and responsibilities, and the list could go on. 
In order for this "primer" to be most effective, it seems that possibly two 
major issues should be addressed, Iflien the question is asked, "^iHiat should we 
evaluate?", the first answer should probably be "the objectives," Being the 
logical starting point for an evaluation, the objectives of a program should 
be adequate descriptors of what the program is all about. Generally, the 
jargon of evaluation labels program objectives as criteria for evaluation 
(Steele, 1970), In many evaluation designs, criteria become the starting 
point for delineating what data is to be collected for the evaluation effort, 

A major evaluation data concern which we have identified in the realm of 
social program evaluation is that relating especially to those non-economic 
variables which have a bearing on social programs. Part of the legacy of 
social program evaluation, and evaluation research in general, has been the 
concern for cost-benefit analysis. Needless to say, the knowledge, attitudes, 
skills, and aspirations of program target audiences are very difficult to 
operationalize in terms of cost-benefit analyses (Rothenberg, 1975), 
Dollars-and-cents figures do not necessarily apply to important program 
considerations such as staff interaction. There is an increasing emphasis 
in social program evaluation tov/ard the measurement of non-economic variables. 
This issue is exemplified through such considerations as using attitudes of 
program recipients as a measure of program effectiveness. 
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What this all seems to point to is an expanded "bag of tricks" for the 
program evaluator. The evaluator must be aware of any type of data collection 
device, no matter what the source, which can aid him in more effectively 
evaluating a program. The evaluator must learn to rely not only on data 
gathered from detailed survey designs, but must also to be able to apply more 
informal techniques such as participant observation (Fry, 1973), Although 
evaluation data coming from many sources may tend to get involved, the chances 
become greater that fewer important aspects of the program will be neglected 
in the examination (Webb, et, al, , 1966; and Moursund, 1973), In short, 
the social program evaluator must be flexible, and always innovative, in 
conducting an evaluation of a social program (Weiss and Rein, 1969; Rippey, 
1973; Deutscher, 197A; Deutscher, 1975; Talmage, 1975; and Ball, 1975), 



The Evaluation Process 

Just as there is no single, accepted outline for the procedure known as 
"the scientific method," the process of evaluation also has no single set of 
steps or procedures which to follow in conducting an evaluation- In the 
jargon of evaluation, such processes may be called various names. In the 
literature, "the process of evaluation" may be subsumed under such labels as 
models, frameworks, or approaches to evaluation (Steele, 1973), Generally,, 
all these terms refer to the same thing, the evaluation process. Rather than 
to advocate any one particular model over another, we have found that there 
are similarities among the models (Wetherill and Buttram, 1976), As such, we 
have identified that the evaluation process encompasses several separate steps. 

Logically, any process, especially the evaluation process, must have both 
a starting point and an ending point. The starting point of the evaluation 
process was identified as beginning somewhere prior to the actual implementa- 
tion of an evaluation design. The ending point was identified as falling soon 
after the evaluation exercise vas completed. These two points in the process 
of . evaluation were considered important enough to be phases in and of then^- 
selves. Between these two starting and ending phases in the process, we 
identified three basic phases which, totaled, represent a five-phase process 
of evaluation. Overall, the process of evaluation as presented in Figure 1 
lists the five elements which are more or less common to all the evaluation 
designs which we examined. 



Program 
Description 

^ ACrtreria 



Formulation^^ 

^ Evidence 



f , t 

I 1 * Gachering\^ 

I I ^ Judgment 

, i , I ' Making \ 

I I t f ^Dissemination 

L 'l J ' 

Figure i. 
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This process of 'evaluation is of course very general in appearance. The 
benefits of this generalized procedure are found mostly in Its flexibility. 
With a general framework such as this, an evaluator may specify within each of 
the basic phases the process of evaluation which is most applicable to the 
social program being evaluated. The details are left up to the evaluator or 
the evaluation team. 

The program descrip tion phase is the beginning phase of the evaluation 
process. During this phase the evaluator describes the program. 
This documientation is an attempt to put down on paper what the program 
IS air about. This serves to direct the evaluator, program staff, 
and appropriate audiences as to the program's primary goals and what 
its courses of action have been; Generally, this first phase consists 
of a written description of the program which serves c7.s a documen- 
tation of the goals and operations of the program. 

The criteria formulation phase focuses the evaluation on specific 
aspects of the program. Ivhile it is important to consider the 
objectives of the program in setting up these criteria, it is almost 
impossible to systematically examine all aspects of the program. 
What is^helpful here is to identify the most critical areas of the 
program's operation. Thege critical areas then become the focus of 
^ . the evaluation. Standards, or acceptable levels of performance., 

must be developed for each of these criteria. Again, the objectives 
of the program cannot be ignored in the establishment of evaluation 
criteria. 

The evidence Rathe ring phase concerns itself with the gathering of 
evaluative data. This phase in the process of evaluation specifies 
the data to be collected for evaluation evidence. Evidence is 
gathered concerning the performance or operation of the program 
according to the evaluation criteria formulated and established in 
the preceding phase. Data is collected during this phase on each of 
the specified evaluation criteria. Construction of instruments and 
appropriate statistical analyses of collected data occur during this 
phase. ^ At this p'oint the evaluation is probably more similar to the 
traditional research investigation than at any othdr phase in the 
evaluation process. The skills and knowledge' required of the 
evaluator relate to the principles and mechanics of research design, 
methodology-, and analysis. 

The iudgment-mnking phase is the most impor-^ant phase in the whole 
evaluation process. Without judgment-making there would be no eval- 
uation. The process of making judgments is usually left up to the 
evaluator. As a starting point for making judgments on the program, 
the process of evaluation requires that the data collected for 
evaluation be compared to the criteria established for evaluation 
and judgments be made on this infomiation regarding the performance 
of the program. Such judgments are usually based on the discrepancy 
between the ideal conditions presented in the criteria and the 
real conditiccii; a indicated by the data colleetion effort. Judgments 
and/or program alternatives are based upon the recognition and 
identifications of such discrepancies. 
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The dissemination phase is the last phase of the evaluation process. 
Very few guidelines or procedures for reporting evaluation results 
have been set down. However, such findings are usually reported hAck 
to clients. Sometimes special contractual - arrangements are made 
to include wider audiences., The use of the evaluative findings depends 
upon the status of the program operation.. However, it is very 
important to remember that evaluation results must be fed back into 
the program^, Only througla such feedback can an evaluation make an 
Impact upon future prograiniiiing and program decisions. 

These then are the five basic phases to the evaluation process. Although 
these five phases are necessary in any evaluation^ additional specifications may 
easily be made within the phases in ^order to customize the process to any rural 
development program under examination. It must also be noted, that thesti phases i 
or any specification thereof, do not necessarily have to be In a straight llneat 
sequence. In other wor.ds, the process of evaluation is not locked into a 1, 

2,3,4,5 sequence* Re-specifications may have to be made during the process 

of evaluation. IVhen such becomes the case, it is necessary to retreat back to an 
earlier phase in the development of the evaluation (dotted arrows, Figure 1.) 
so that the program may be adequately evaluated. Re-documentation of the program, 
or respecif ication of the criteria, may have to be done, especially due to the 
often, dynamic nature of development projects. 

Problems and Pitfalls 

Needless to say, program evaluation is a dangerous thing. Evaluation has 
in its power the fate of the program* Misunderstandings as to the conduct of 
evaluation may often lead to splits in program staff sometimes going in as 
many as three directions, including administrators, evaluators, and field 
workers, all on different sides of the issue (Glennan, 1972; Suchmah, 1971; 
and Wright and Hyman, 1964). Until such a time as the process of evaluation is 
more completely understood by all rural development personnel, any evaluator or 
potential evaluator of a rural development project must be aware of the probletns 
and pitfalls inherent in the evaluation research enterprise, A brief list of 
some of the things an evaluator should be concerned with in order to avoid 
problems might look something like this: 

(1) The ethical responsibilities of an evaluator should be beyond reproach 
(Clinton, 1976). 

(2) The responsibility for program evaluation should be delegated to 
one person. This would include the commensurate authority to best be able to 
implement evaluation plans (Bernstein and Freeman, 1975; and Cosby, et* al, , 
1976). 

(3) The total staff of the program should be informed about x^hat is 
going on iii an evaluation. Evaluation plans must be communicated to all 
relevant program personnel, so there are no surprises with respect to what is 
going to happen for the evaluation (Stuff lebeam, et, al, , 1971; and Moursund, 
1973), 

(4) Reliance on a single source of data as evidence for evaluation should 
be scrutinized most closely. Data from many different sources usually lends 
more validity to evaluation evidence (Webb, et, al. , 1966; and Moursund, 1973), 

(5) The evaluator is usually treated as in intruder Into the program. As 
such, he must be aware of the pressures and constraints on the program being 
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evaluated. The evaluator must be a diplomat (Longood and Simmel, 1972; Weiss, 
1972; and Bogdon, 1975), 

(6) The evaluator along with program staff, must get together early in 
the process of evaluation in order to negotiate the terms of evaluation. Areas, 
of such negotiation include: ways and means of dissemination, access to project 
personnel and records, audiences for the evaluation results, roles and respon- 
sibilities of the evaluator, etc, (Ber^edict, 1973; Burton, 1976; Provus, 1971; 
and United Nations, 1962) • 

(7) The evaluator, as well as th'e evaluation, must be flexible* Usually, 
the dynamic nature of rural development ptograms precludes a strict reliance 

oil detailed evaluation schedules. To a certain degree, evaluation .plans should 
not act as a straight-jacket to evaluation (Suchman, 1967), 

(8) Under most circumstances, (Bvaluators ought not to get embroiled with 
actual decision-^making functions in a program. The typical role of the evaluator 
might be that of spelling out options for program alternatives but not that of 
making actual decisions exclusive of program staff (Moursund, 1973), 

(9) The evaluator should be committed to evaluation. His attitude toward 
the evaluation process should be positive. This is in contrast to his attitude 
toward the program which should be unbiased (Bogdon, 1975; and Suchman, 1967). 

(10) Evaluation feedback should be given to appropriate program personnel 
as the evaluation progresses. This does not mean, however, that evaluation 
results should be given out prematurely (Stuff lebeam, et. al. , 1971; and Moursund, 
19 73). 

(11) If possible, get someone to react to your evaluation plans before 
implementing the plans. But know when to stop asking for advice (Benedict, 1973; 
Burton, 19 76; and Weiss, 1972). 

The program evaluator is a potentially powerful yet potentially dangerous 
role. An evaluator must be a specialized generalist. The evaluator must have 
a knowledge of basic research methods and statistics, techniques of evaluation 
(the evaluator 's bag of tricks), a knowledge of program development techniques 
(including proposal writing, budgeting, staffing, program planning, program 
implementation, and above all, the ability to define goals and objectives), 
and knowledge of both the social system in which the program is operating as 
well as interpersonal dynamics internal to the program. 

In sum, a program evaluator, as well as program evaluation, is in a unique 
position to aid rural development. Rural development programs which have 
evaluation built into the program not only have a higher likelihood of success, 
but are also good examples for subsequent development programs. There is no 
magic formula for evaluation, nor is any program evaluator a magician. The 
fact remains, however, that program evaluation has become almost mandatory not 
only for rural development programs, but also for social action programs in 
general. The more that is known about the evaluative research enterprise, the 
better it will be for the acceptance of evaluative results, hence program 
improvement. When this occurs, evaluative research will not be the "shady" 
enterprise which many people suggest that it is. 
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